Graduate Student Perspectives on Replication in HCI Research
•Download as PPTX, PDF•
0 likes•15 views
The document summarizes the perspectives of graduate students in SIGCHI on replicating studies. A survey found that 17% of students had replicated a study before, while 83% had not. 38% said they would be willing to replicate in the future, while 62% said they would not. Several students felt there is no reward for replicating since it is not novel work, and reviewers often criticize submissions for not being novel enough. The document suggests the current incentives in SIGCHI discourage replication work.
Recommended
Recommended
More Related Content
Similar to Graduate Student Perspectives on Replication in HCI Research
Similar to Graduate Student Perspectives on Replication in HCI Research (20)
More from Michael Bernstein
More from Michael Bernstein (11)
Recently uploaded
Recently uploaded (20)
Graduate Student Perspectives on Replication in HCI Research
- 1. RepliCan’t Graduate Student Perspectives Michael Bernstein MIT Computer Science and Artificial Intelligence Laboratory msbernst@mit.edu | @msbernst mit human-computer interaction
- 2. Speaking for every graduate student in SIGCHI, I can say one thing:
- 3. I can’t speak for every graduate student in SIGCHI.
- 4. My Unassailable, Extremely ScientificData Collection Protocol: Survey of CHI student volunteers, CHI-students ACM listserv, and snowballed recruitment through Facebook and Twitter. N=93 responses
- 5. Have you ever replicateda study or system? 17% yes, 83% no
- 6. Do you ever plan to replicatea study or system? 38% “Hell yes”, 62% “Hell no”
- 7. Set Yourself Apart “ The point of research is to come up with exciting ideas that solve problems. Not copy others’ work. ”
- 8. Set Yourself Apart “ ” I'm more creative than that.
- 9. There’s No Reward “ New studies confirming old studies have no chance of publication. ”
- 10. There’s No Reward Reviewers […] didn't feel replication was necessary even though the original study was specific to a single company. “ ”
- 11. There’s No Reward I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”. “ ”
- 12. There’s No Reward I very frequently see reviewers criticize submissions for presenting results that are “not novel” or “have already been shown”. “ ”
- 13. Responding to Incentives Open access and replication. A true scientist’s ideals, but see:The grad student must conform. “ ”
- 14. Haikus studies should break ground replication wastes our timelet's find new problems “ ”
- 15. Haikus think analyzing CMC is tough? try it reproducibly! “ ”
- 16. Haikus repeat to be sure we stand on giants’ shoulders but do so on faith “ ”
- 17. 83% have not 62% will not (But we’ll need to replicate the study to be sure.)
- 19. Why? “ repeat to be sure we stand on giant's shoulders but do so on faith ”
- 20. Why? “ Replication is a critical component of scientific research, and it should be encouraged and rewarded. The lack of it is detrimental to the scientific soundness of our discipline. ”
- 21. Why? “ we lack the time for replication of studies just review strictly ”
- 22. Why? “ think analyzing CMC is tough? try it reproducibly! ”
- 23. Why? “ It seems like the best outcome for a replication, rather than success, is actually a refutation of the original study. ”
- 24. Why? “ Because its's a waste of time: HCI studies are so small, I know they surely WON'T replicate, so why bother! ”
- 25. Why? “ CHI is too cutting edge for things like replication, or good science, or careful analysis, or the humility to accept that other topics besides Fitts' Law deserve dozens of nearly-identical studies. ”
- 27. Why? “ I do not intend on taking the risk of replicating some of my favorite works unless I see evidence that the CHI community supports such a thing. ”
- 28. There’s No Reward Case A. Confirmation of the earlier results (very boring)Case B. Conflict with earlier results (unpublishable problem) “ ”
Editor's Notes
- Interfaces that embed crowd contributions to support complex and novel interactive systems.-------------Bring in the simplex? Thread through, come back to it from time to time.Soylent. Emphasize that we’re not throwing away AI, not throwing away the user, …Explain FeedMe better via the simplex?Insights from designing two different families of interfaces --- generalize lessons? Motivations. Design differences. Commonalities: small bits of effort? At end of talk.- small bits of effortMost of this has examined moving things toward crowd. But, we need a better sense of when to do that, and when not to. Feed back into traditional We need to understand in order to build. Not just the user interface, but the crowd.Embiggen the gray from data mining. Move data mining to future work?Introduce points one and two, these are the systems I’m going to talk about in this talk. Then, at the end, say there are two avenues for future work. 1) realtime, 2) data thing thing embiggen thing.Data-driven?Less cadence at the end of Soylent.Transition between Collabio and FeedMe needs work.Second half was too fast-paced? Video especially?Stupid PhD student.25 min soylent, 15 min Don’t make crazy animation w/ ShortnTry to save the Soylent demo for later. Figure out a way to convey the concept without the early handwaved demo.“Pa-nah-cea”Finish the transition. There are lots of applications where you can't use crowds. One of those is when you need socially sourced information. We have to develop techniques for this